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Abstract 


This  report  details  a  statistical  approach  to  the  comparison  of  results  from  an  international 
study  in  which  Navier-Stokes  con5)utational  techniques  were  applied  to  a  complex  flow  field  with 
highly  separated  flow  about  a  fixed  geometry.  The  traditional  method  for  evaluating  flow-field 
computations  is  for  the  engineer  to  visually  inspect  the  results,  make  qualitative  comparisons,  and 
draw  conclusions — a  process  that  can  be  effective  for  a  limited  number  of  data  sets.  In  this  study, 
the  number  of  data  sets  was  so  large  as  to  make  the  evaluation  and  comparison  of  the  results  a 
formidable  task.  In  response,  a  statistical  approach  widi  emphasis  on  data  visualization  was 
developed.  The  procedure  extends  directly  to  higher  dimensions,  and  is  of  particular  value  when 
a  large  number  of  data  sets  are  to  be  compared  and  an  inpartial  quantitative  assessment  is  sought. 
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1.  INTRODUCTION 


This  report  discusses  the  development  and  application  of  a  statistical  analysis  approach  to 
the  evaluation  of  some  results  of  a  collaborative  study  carried  out  under  the  auspices  of  The 
Technical  Cooperation  Program  (TTCP)  with  participants  from  Canada,  the  United  Kingdom,  and 
the  United  States.  The  purpose  of  the  study  was  to  apply  Navier-Stokes  computational  techniques 
to  a  complex  flow  field  with  highly  separated  flow  about  a  fixed  missile  geometry  to  evaluate  the 
predictive  technology. 

The  computational  study  encompassed  8  computer  codes,  9  turbulent  viscous  models,  several 
laminar  viscous  results,  as  many  as  15  grid  resolutions,  and  6  test  cases.  This  produced  more  than 
90  test-case  results  for  evaluation  with  respect  to  each  other  and  to  effects  of  grid  resolution,  choice 
of  turbulence  model,  and  computational  technique. 

A  Schlieren  photograph  showing  the  test  model  mounted  in  the  wind  tunnel  for  a  Mach  3.5  flow 
condition  and  8-deg  angle  of  incidence  is  shown  in  Figure  1.  The  bow  shock  and  lee-side  separated 
vortical  flow  are  clearly  visible.  In  Figure  2,  experimental  measurements  of  surface  pressure  taken  at 
eight  axial  locations,  x/d,  are  plotted  along  with  the  computed  values  from  laminar  and  turbulent 
viscous  models.  The  computed  results  show  generally  close  agreement  with  the  experiment  on 
the  windward  side  (0  <  ^  <  90  deg)  and  disagreement  on  the  leeward  side  (90  <  (p  <  180  deg). 
The  lee-side  separated  flow  is  the  critical  region  for  evaluating  the  predictive  capability  of  the 
computational  procedures. 


Figure  1.  Schlieren  photograph  of  model  mounted  in  wind  tunnel;  Mach  3.5,  angle  of  incidence 
8  deg. 

The  traditional  method  for  evaluating  flow-field  computations  is  for  the  engineer  to  visually 
inspect  the  results,  make  qualitative  comparisons,  and  draw  conclusions — a  process  that  can  be 
quite  satisfactory  for  a  limited  number  of  data  sets.  However,  for  this  study,  the  number  of  data 
sets  was  so  large  as  to  make  the  evaluation  and  comparison  of  the  results  a  formidable  task,  and  it 
was  decided  to  explore  the  contribution  that  statistical  data  analysis  techniques  might  be  able  to 
make  in  the  evaluation  process. 
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Figure  2.  Surface  pressure  Cp  vs.  circumferential  position  <p;  Mach  2.5,  angle  of  incidence 
14  deg.  Comparison  between  computation  and  experiment. 
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2.  STATISTICAL  APPROACH 


The  wide  availability  of  powerful  and  affordable  computing  resources  has  impacted  the  way  in 
which  data  analysis  is  conducted  at  a  fundamental  level,  and  expressions  such  as  ‘^exploratory  data 
analysis”  and  “data  visualization”  have  made  their  way  into  the  statistical  lexicon.  Visualization 
of  quantitative  data  is  an  active  area  of  research,  with  professional  journals  and  books  devoted 
exclusively  to  the  topic  (Tufte  1983,  1990;  Cleveland  1993;  DuToit,  Steyn,  and  Stumpf  1986).  The 
basic  idea  is  that  data,  if  properly  collected  and  effectively  portrayed,  can  often  speak  directly  to 
the  subject-matter  expert  without  resorting  to  a  plethora  of  statistical  jargon  and  assumptions. 
This  is  the  approach  undertaken  in  this  section;  a  subset  of  TTCP  test-case  results  is  used  for 
illustrative  purposes. 

2.1  Data  Structure 

A  set  of  experimental  data  was  chosen  to  serve  as  a  baseline  against  which  all  the  flowfield 
predictors  would  be  compared.  The  experimental  data  pertinent  to  this  investigation  consist  of 
measurements  of  surface  pressure  taken  at  41  circumferential  positions,  0,  about  each  of  9  axial 
locations,  x/d,  for  a  test  model  such  as  that  shown  in  Figure  1. 

The  technique  involves  the  comparison  of  two  data  sets — the  experimental  (or  baseline)  data 
and  a  set  of  computed  results.  Both  data  sets  are  three-dimensional,  consisting  of  the  triples:  (axial 
location,  circumferential  position,  surface  pressure).  To  facilitate  discussion,  the  experimental  data 
are  denoted  as 

Vij  2  =  3.5(1)11.5,  j  =  0(4.5)180,  (1) 

where  the  subscript  i  indexes  axial  location,  x/d;  the  subscript  j  indexes  circumferential  position, 
0;  and  Yij  is  the  measured  surface  pressure.  The  computed  value  is  denoted  as 

Xj  k  =  1(1)9,  (2) 

where  subscripts  i  and  j  are  the  same  as  for  the  experimental  data;  the  superscript  k  indexes  the 
data  sets  chosen  for  analysis. 

2.2  Error  Analysis 

If  we  define  the  difference  between  a  computed  value  of  surface  pressure  and  the  corresponding 
experimental  value  as  error,  then 

E^  =  X^-Yij  (3) 

is  the  error  at  location  (2,  j)  for  data  set  k.  A  three-dimensional  contour  plot  provides  an  informative 
display  of  error  over  the  entire  axial-circumferential  grid;  such  a  plot  for  the  data  set  corresponding 
to  A:  =  1  is  shown  in  Figure  3. 

The  approximately  planar  surface  in  Figure  3  represents  the  windward  region  where  experi¬ 
mental  and  computed  values  are  in  reasonably  good  agreement.  The  projection  of  the  surface  onto 
the  horizontal  plane  further  aids  in  the  definition  of  this  region;  for  data  set  A:  =  1,  the  substantial 
irregularities  occur  on  the  leeward  side,  corresponding  to  values  of  (f)  greater  than  90  deg. 

The  highly  irregular  contour  surface  in  Figure  3  provides  convincing  evidence  of  the  need  for 
further  data  reduction  before  the  simultaneous  comparison  of  several  flowfield  calculations  can  be 
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Error 


Figure  3.  Contour  surface  of  error  for  data  set  k=L 


effectively  undertaken.  A  further  data  summary  using  box  and  whisker  plots  is  shown  in  Figure  4. 
To  construct  a  box  and  whisker  plot,  the  errors  over  the  entire  grid  are  combined  and  ranked  in 
value  from  smallest  to  largest.  The  median,  or  middle,  value  is  visually  represented  by  the  dot  (•) 
inside  the  boxes  in  Figure  4;  the  top  and  bottom  of  the  box  mark  the  75th  and  25th  percentiles, 
respectively,  of  the  ranked  errors;  the  whiskers  are  vertical  lines  that  terminate  at  the  extreme  values 
of  the  data  set.  The  width  of  the  box  holds  no  significance.  These  five-number  data  summaries 
provide  a  compact  description  of  how  the  errors  are  distributed  over  the  entire  grid,  and,  as  can  be 
seen,  facilitate  comparison  across  several  data  sets. 

The  perfect  case,  where  computed  and  experimental  values  coincide  at  every  location  (ij), 
would  cause  the  box  and  whisker  plot  to  collapse  to  a  single  value — ^zero.  Failing  that,  a  thin  box 
with  short  whiskers  indicates  good  overall  agreement.  A  cursory  glance  at  Figure  4  suggests  that 
data  sets  3  and  4  may  be  closest  to  this  ideal,  while  data  set  8  appears  in  some  difficulty.  Since  the 
whiskers  terminate  at  the  maximum  and  minimum  error  values,  a  spurious,  or  outlying,  observation 
could  distort  the  whisker  length  and  mislead  the  viewer.  The  data  sets  in  Figure  4  were  inspected 
and  found  to  be  free  from  outliers — a  precautionary  procedure  that  should  be  routinely  performed. 

The  statistical  graphics  in  Figures  3-4,  while  enlightening,  suggestive,  and  highly  appropriate 
for  an  initial  screening,  are  not  entirely  adequate  for  an  impartial  assessment  of  which  error  sets 
represent  “closest”  agreement  between  experiment  and  model.  Clearly,  we  would  like  the  errors  to 
be  tightly  clustered  about  zero.  This  would  be  reflected  in  a  location  parameter  (a  mean  or  median) 
and  a  dispersion  parameter  (a  standard  deviation  or  interquantile  range)  both  close  to  zero.  This 
suggests  that  an  attempt  to  formally  rank  the  effectiveness  of  the  computation  procedures  as 
revealed  through  the  error  sets  k  =  1,2,  ...,9,  should  involve  a  statistic  that  includes  a 

measure  of  both  location  and  dispersion.  And  this  is  exactly  what  was  done. 
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Error 


Figure  4.  Box  and  whisker  plot  of  errors;  k=lj  2,  . . . ,  9, 

We  chose  as  a  statistic  the  distance  between  the  point  determined  by  the  sample  mean  and 
standard  deviation  [xk^Sk)  and  the  origin.  Recall  that  the  origin  (0,0)  corresponds  to  perfect 
agreement  between  calculation  and  experiment.  The  distances 

/c  =  l,2,  ...,9  (4) 

were  determined,  and  the  ranking  induced  under  this  procedure  in  terms  of  the  index  A:  is  as  follows: 

4^3^2^7-<5x6^1^9-<8.  (5) 

Expression  (5)  asserts  that  the  set  of  computed  data  corresponding  to  k=A  is  in  closest  agree¬ 
ment  with  the  experimental  measurements;  data  set  A:=3  is  second  closest,  ...,  data  set  A:=8  is 
furthest  away. 

In  Figure  5,  the  sample  mean  and  standard  deviation  [xk^Sk)  for  each  of  the  9  data  sets  are 
plotted  by  case  number  k.  More  complicated  ranking  procedures  are,  of  course,  possible.  An 
obvious  modification  would  be  to  form  a  weighted  combination  of  mean  and  standard  deviation, 
but  to  do  so  without  a  compelling  reason  serves  no  practical  purpose,  and  was  not  undertaken  for 
these  data. 


2.3  Summary 

The  results  of  the  direct  ranking  procedure  are  altogether  consistent  with  the  preliminary  graph¬ 
ics.  This  is  reassuring.  A  surprise  at  this  stage  usually  means  that  an  error  in  logic  or  calculation 
is  still  at  large.  As  mentioned  earlier,  and  as  seen  in  Figure  2,  the  flowfield  predictors  seem  to 
experience  the  most  difficulty  for  larger  circumferential  values,  so  focusing  on  that  subregion  rather 
than  the  entire  flowfield  may  be  more  appropriate. 

For  the  statistician,  failure  to  account  for  error  in  the  experimental  measurements  Yij  is  trou¬ 
blesome.  The  statistician’s  model  for  experimental  data  is  usually  of  the  form:  observed  = 
actual  Yij  +  error  eij.  Engineering  experience  leaves  the  fluid  dynamicist  convinced  that  concern 
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Figure  5.  Scatterplot  of  mean  and  standard  deviation  for  error  sets  k=l,  2,  . . . ,  9. 

over  this  point  is  unwarranted.  Moreover,  in  this  instance,  only  a  single  set  of  experimental  data 
is  available  and  all  the  flowfield  predictors  are  deterministic,  so  the  error,  whatever  its  magnitude, 
remains  forever  confounded  with  the  recorded  observations. 


3.  APPLICATION  TO  TTCP  DATA 


The  approach  developed  in  section  2  was  applied  to  output  from  four  turbulence  models:  the 
Baldwin-Lomax(BL),  Baldwin-Barth(BB),  k-omega(kw),  and  k-epsilon(ke),  all  using  the  NPARC 
computational  code  (Cooper  and  Sirbaugh  1989).  The  theory  and  implementation  of  these  models 
is  well  documented  and  is  apart  from  our  considerations  here. 

The  difficulty  of  visual  comparison  between  experiment  and  computation  over  the  four  models 
is  convincingly  illustrated  in  Figure  6.  Only  four  axial  stations  are  shown,  but,  even  from  that  small 
number,  the  difficulty  facing  the  engineer  attempting  to  rank  the  procedures  is  readily  apparent. 

The  statistical  approach  was  applied  to  the  entire  data  sets:  41  circumferential  locations  at 
each  of  14  axial  stations  for  the  4  turbulence  models.  The  box  and  whisker  plot  visualization  is 
shown  in  Figure  7.  The  BL  result  has  the  shortest  whiskers,  indicating  the  smallest  excursion  from 
the  experimental  data.  The  mean  error  for  all  the  data  sets  is  close  to  zero,  with  little  apparent 
advantage  for  one  turbulence  model  over  another.  The  results  are  shown  in  a  scatterplot  format 
in  Figure  8.  Again,  the  BL  model  seems  to  have  the  advantage;  however,  as  shown  in  Table  1,  the 
level  of  disagreement  between  the  four  turbulence  models  is  very  small. 

The  formal  ranking  is  BL  -I  kw  ^  ke  -<  BB,  but,  importantly,  the  small  numerical  differences 
listed  in  Table  1  hold  substantial  practical  significance  for  the  engineer.  The  four  turbulence  models 
vary  widely  in  complexity  and  computer  resource  requirements.  The  most  computer  efficient  model 
is  the  algebraic  model  BL,  followed  by  the  BB  model.  The  two-equation  models,  kw  and  ke,  are 
more  complex  and  require  significantly  greater  computer  resources  than  either  the  BB  or  BL  model. 
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Figure  8.  Scatterplot  of  errors  for  BB,  BL,  ke,  and  kw  data  sets;  Mach  3.5,  angle  of  incidence 
14  deg. 

Table  1.  Distance  Measurements  for  (x^  s) 

case:  BB  BL  ke  kw 

distance:  0.01257  0.01125  0.01149  0.01131 


4.  CONCLUSION 

A  statistical  approach,  with  emphasis  on  data  visualization,  has  been  developed  to  assist  the 
engineer  in  the  comparison  of  computer  models  intended  to  predict  surface  pressure  about  a  phys¬ 
ical  model  in  a  complex  flow  fleld.  The  approach  was  applied  to  the  evaluation  of  computational 
results  obtained  from  four  distinct  turbulence  models  and  showed  that  the  use  of  more  sophisti¬ 
cated  computer-intensive  turbulence  models  did  not  provide  correspondingly  superior  results.  The 
statistical  analysis  procedure  extends  directly  to  higher  dimensions  and  should  be  particularly  valu¬ 
able  when  a  large  number  of  data  sets  are  to  be  compared,  and  when  an  impartial  quantitative 
assessment  is  sought. 
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